A practical guide to EEG hyperscanning in joint action research: from motivation to implementation.

Developments in cognitive neuroscience have led to the emergence of hyperscanning, the simultaneous measurement of brain activity from multiple people. Hyperscanning is useful for investigating social cognition, including joint action, because of its ability to capture neural processes that occur within and between people as they coordinate actions toward a shared goal. Here, we provide a practical guide for researchers considering using hyperscanning to study joint action and seeking to avoid frequently raised concerns from hyperscanning skeptics. We focus specifically on Electroencephalography (EEG) hyperscanning, which is widely available and optimally suited for capturing fine-grained temporal dynamics of action coordination. Our guidelines cover questions that are likely to arise when planning a hyperscanning project, ranging from whether hyperscanning is appropriate for answering one's research questions to considerations for study design, dependent variable selection, data analysis and visualization. By following clear guidelines that facilitate careful consideration of the theoretical implications of research design choices and other methodological decisions, joint action researchers can mitigate interpretability issues and maximize the benefits of hyperscanning paradigms.

The spontaneous emergence of rhythmic coordination in turn taking.

Turn-taking is a feature of many social interactions such as group music-making, where partners must alternate turns with high precision and accuracy. In two studies of musical rhythm coordination, we investigated how joint action partners learn to coordinate the timing of turn-taking. Musically inexperienced individuals learned to tap at the rate of a pacing cue individually or jointly (in turn with a partner), where each tap produced the next tone in a melodic sequence. In Study 1, partners alternated turns every tap, whereas in Study 2 partners alternated turns every two taps. Findings revealed that partners did not achieve the same level of performance accuracy or precision of inter-tap intervals (ITIs) when producing tapping sequences jointly relative to individually, despite showing learning (increased ITI accuracy and precision across the experiment) in both tasks. Strikingly, partners imposed rhythmic patterns onto jointly produced sequences that captured the temporal structure of turns. Together, learning to produce novel temporal sequences in turn with a partner appears to be more challenging than learning to produce the same sequences alone. Critically, partners may impose rhythmic structures onto turn-taking sequences as a strategy for facilitating coordination.

Is neural entrainment to rhythms the basis of social bonding through music?

Music uses the evolutionarily unique temporal sensitivity of the auditory system and its tight coupling to the motor system to create a common neurophysiological clock between individuals that facilitates action coordination. We propose that this shared common clock arises from entrainment to musical rhythms, the process by which partners' brains and bodies become temporally aligned to the same rhythmic pulse.

Behavioral and Neural Dynamics of Interpersonal Synchrony Between Performing Musicians: A Wireless EEG Hyperscanning Study.

Interpersonal synchrony refers to the temporal coordination of actions between individuals and is a common feature of social behaviors, from team sport to ensemble music performance. Interpersonal synchrony of many rhythmic (periodic) behaviors displays dynamics of coupled biological oscillators. The current study addresses oscillatory dynamics on the levels of brain and behavior between music duet partners performing at spontaneous (uncued) rates. Wireless EEG was measured from N = 20 pairs of pianists as they performed a melody first in Solo performance (at their spontaneous rate of performance), and then in Duet performances at each partner's spontaneous rate. Influences of partners' spontaneous rates on interpersonal synchrony were assessed by correlating differences in partners' spontaneous rates of Solo performance with Duet tone onset asynchronies. Coupling between partners' neural oscillations was assessed by correlating amplitude envelope fluctuations of cortical oscillations at the Duet performance frequency between observed partners and between surrogate (re-paired) partners, who performed the same melody but at different times. Duet synchronization was influenced by partners' spontaneous rates in Solo performance. The size and direction of the difference in partners' spontaneous rates were mirrored in the size and direction of the Duet asynchronies. Moreover, observed Duet partners showed greater inter-brain correlations of oscillatory amplitude fluctuations than did surrogate partners, suggesting that performing in synchrony with a musical partner is reflected in coupled cortical dynamics at the performance frequency. The current study provides evidence that dynamics of oscillator coupling are reflected in both behavioral and neural measures of temporal coordination during musical joint action.

The sound of silence: an EEG study of how musicians time pauses in individual and joint music performance.

Pauses are an integral feature of social interaction. Conversation partners often pause between conversational turns, and musical co-performers often pause between musical phrases. How do humans coordinate the duration of pauses to ensure seamless interaction? A total of 40 trained pianists performed a simple melody containing fermatas (notated expressive pauses of unspecified duration) first alone (Solo) and then with a partner (Duet) while electroencephalography (EEG) was recorded. As predicted, Duet partners' tone onset synchrony was reduced for tones following pauses. Pauses were shorter in Duet relative to Solo performance, and synchrony of partners' Duet tone onsets was enhanced for tones following shorter pauses. EEG analysis revealed classic signatures of action preparation during pauses, namely decreases in the power of cortical beta oscillations (13-30 Hz, event-related desynchronization ERD). Beta ERD did not differ between pauses in Solo and Duet performance, but was enhanced for shorter relative to longer pauses, suggesting that reduced pause durations in Duet performance facilitated a neural state of enhanced action readiness. Together these findings provide novel insight into behavioural strategies by which musical partners resolve coordination challenges posed by expressive silence, and capture a clear neural signature of action planning during time-varying silences in natural music performance.

Rhythm Complexity Modulates Behavioral and Neural Dynamics During Auditory-Motor Synchronization.

We addressed how rhythm complexity influences auditory-motor synchronization in musically trained individuals who perceived and produced complex rhythms while EEG was recorded. Participants first listened to two-part auditory sequences (Listen condition). Each part featured a single pitch presented at a fixed rate; the integer ratio formed between the two rates varied in rhythmic complexity from low (1:1) to moderate (1:2) to high (3:2). One of the two parts occurred at a constant rate across conditions. Then, participants heard the same rhythms as they synchronized their tapping at a fixed rate (Synchronize condition). Finally, they tapped at the same fixed rate (Motor condition). Auditory feedback from their taps was present in all conditions. Behavioral effects of rhythmic complexity were evidenced in all tasks; detection of missing beats (Listen) worsened in the most complex (3:2) rhythm condition, and tap durations (Synchronize) were most variable and least synchronous with stimulus onsets in the 3:2 condition. EEG power spectral density was lowest at the fixed rate during the 3:2 rhythm and greatest during the 1:1 rhythm (Listen and Synchronize). ERP amplitudes corresponding to an N1 time window were smallest for the 3:2 rhythm and greatest for the 1:1 rhythm (Listen). Finally, synchronization accuracy (Synchronize) decreased as amplitudes in the N1 time window became more positive during the high rhythmic complexity condition (3:2). Thus, measures of neural entrainment corresponded to synchronization accuracy, and rhythmic complexity modulated the behavioral and neural measures similarly.

Synchronizing MIDI and wireless EEG measurements during natural piano performance.

Although music performance has been widely studied in the behavioural sciences, less work has addressed the underlying neural mechanisms, perhaps due to technical difficulties in acquiring high-quality neural data during tasks requiring natural motion. The advent of wireless electroencephalography (EEG) presents a solution to this problem by allowing for neural measurement with minimal motion artefacts. In the current study, we provide the first validation of a mobile wireless EEG system for capturing the neural dynamics associated with piano performance. First, we propose a novel method for synchronously recording music performance and wireless mobile EEG. Second, we provide results of several timing tests that characterize the timing accuracy of our system. Finally, we report EEG time domain and frequency domain results from N=40 pianists demonstrating that wireless EEG data capture the unique temporal signatures of musicians' performances with fine-grained precision and accuracy. Taken together, we demonstrate that mobile wireless EEG can be used to measure the neural dynamics of piano performance with minimal motion constraints. This opens many new possibilities for investigating the brain mechanisms underlying music performance.

Musicians' Natural Frequencies of Performance Display Optimal Temporal Stability.

Many human action sequences, such as speaking and performing music, are inherently rhythmic: Sequence events are produced at quasi-regular temporal intervals. A wide range of interindividual variation has been noted in spontaneous production rates of these rhythmic action sequences. Dynamical theories of motor coordination suggest that individuals spontaneously produce rhythmic sequences at a natural frequency characterized by minimal energy expenditure and maximal temporal stability, relative to other frequencies. We tested this hypothesis by comparing the temporal variability with which musicians performed rhythmic melodies at their natural spontaneous rate with variability in their performances at faster and slower rates. Musicians' temporal variability was lowest during performances at their spontaneous rate; in addition, performers' tempo drift during trials at other rates showed bias toward their spontaneous rate. This study provides the first direct evidence that spontaneous rates of motor coordination represent optimally stable natural frequencies of endogenous rhythms.

Amplitude envelope correlations measure synchronous cortical oscillations in performing musicians.

A major question facing cognitive neuroscience is measurement of interbrain synchrony between individuals performing joint actions. We describe the application of a novel method for measuring musicians' interbrain synchrony: amplitude envelope correlations (AECs). Amplitude envelopes (AEs) reflect energy fluctuations in cortical oscillations over time; AE correlations measure the degree to which two envelope fluctuations are temporally correlated, such as cortical oscillations arising from two individuals performing a joint action. Wireless electroencephalography was recorded from two pianists performing a musical duet; an analysis pipeline is described for computing AEs of cortical oscillations at the duet performance frequency (number of tones produced per second) to test whether these oscillations reflect the temporal dynamics of partners' performances. The pianists' AE correlations were compared with correlations based on a distribution of AEs simulated from white noise signals using the same methods. The AE method was also applied to the temporal characteristics of the pianists' performances, to show that the observed pair's AEs reflect the temporal dynamics of their performance. AE correlations offer a promising approach for assessing interbrain correspondences in cortical activity associated with performing joint tasks.

Tapping Into Rate Flexibility: Musical Training Facilitates Synchronization Around Spontaneous Production Rates.

The ability to flexibly adapt one's behavior is critical for social tasks such as speech and music performance, in which individuals must coordinate the timing of their actions with others. Natural movement frequencies, also called spontaneous rates, constrain synchronization accuracy between partners during duet music performance, whereas musical training enhances synchronization accuracy. We investigated the combined influences of these factors on the flexibility with which individuals can synchronize their actions with sequences at different rates. First, we developed a novel musical task capable of measuring spontaneous rates in both musicians and non-musicians in which participants tapped the rhythm of a familiar melody while hearing the corresponding melody tones. The novel task was validated by similar measures of spontaneous rates generated by piano performance and by the tapping task from the same pianists. We then implemented the novel task with musicians and non-musicians as they synchronized tapping of a familiar melody with a metronome at their spontaneous rates, and at rates proportionally slower and faster than their spontaneous rates. Musicians synchronized more flexibly across rates than non-musicians, indicated by greater synchronization accuracy. Additionally, musicians showed greater engagement of error correction mechanisms than non-musicians. Finally, differences in flexibility were characterized by more recurrent (repetitive) and patterned synchronization in non-musicians, indicative of greater temporal rigidity.

Knowing When Help Is Needed: A Developing Sense of Causal Complexity.

Research on the division of cognitive labor has found that adults and children as young as age 5 are able to find appropriate experts for different causal systems. However, little work has explored how children and adults decide when to seek out expert knowledge in the first place. We propose that children and adults rely (in part) on "mechanism metadata," information about mechanism information. We argue that mechanism metadata is relatively consistent across individuals exposed to similar amounts of mechanism information, and it is applicable to a wide range of causal systems. In three experiments, we show that adults and children as young as 5 years of age have a consistent sense of the causal complexity of different causal systems, and that this sense of complexity is related to decisions about when to seek expert knowledge, but over development there is a shift in focus from procedural information to internal mechanism information.

Endogenous rhythms influence interpersonal synchrony.

Interpersonal synchrony, the temporal coordination of actions between individuals, is fundamental to social behaviors from conversational speech to dance and music-making. Animal models indicate constraints on synchrony that arise from endogenous rhythms: Intrinsic periodic behaviors or processes that continue in the absence of change in external stimulus conditions. We report evidence for a direct causal link between endogenous rhythms and interpersonal synchrony in a music performance task, which places high demands on temporal coordination. We first establish that endogenous rhythms, measured by spontaneous rates of individual performance, are stable within individuals across stimulus materials, limb movements, and time points. We then test a causal link between endogenous rhythms and interpersonal synchrony by pairing each musician with a partner who is either matched or mismatched in spontaneous rate and by measuring their joint behavior up to 1 year later. Partners performed melodies together, using either the same or different hands. Partners who were matched for spontaneous rate showed greater interpersonal synchrony in joint performance than mismatched partners, regardless of hand used. Endogenous rhythms offer potential to predict optimal group membership in joint behaviors that require temporal coordination.

Temporal coordination in joint music performance: effects of endogenous rhythms and auditory feedback.

Many behaviors require that individuals coordinate the timing of their actions with others. The current study investigated the role of two factors in temporal coordination of joint music performance: differences in partners' spontaneous (uncued) rate and auditory feedback generated by oneself and one's partner. Pianists performed melodies independently (in a Solo condition), and with a partner (in a duet condition), either at the same time as a partner (Unison), or at a temporal offset (Round), such that pianists heard their partner produce a serially shifted copy of their own sequence. Access to self-produced auditory information during duet performance was manipulated as well: Performers heard either full auditory feedback (Full), or only feedback from their partner (Other). Larger differences in partners' spontaneous rates of Solo performances were associated with larger asynchronies (less effective synchronization) during duet performance. Auditory feedback also influenced temporal coordination of duet performance: Pianists were more coordinated (smaller tone onset asynchronies and more mutual adaptation) during duet performances when self-generated auditory feedback aligned with partner-generated feedback (Unison) than when it did not (Round). Removal of self-feedback disrupted coordination (larger tone onset asynchronies) during Round performances only. Together, findings suggest that differences in partners' spontaneous rates of Solo performances, as well as differences in self- and partner-generated auditory feedback, influence temporal coordination of joint sensorimotor behaviors.

Pathways to seeing music: enhanced structural connectivity in colored-music synesthesia.

Synesthesia, a condition in which a stimulus in one sensory modality consistently and automatically triggers concurrent percepts in another modality, provides a window into the neural correlates of cross-modal associations. While research on grapheme-color synesthesia has provided evidence for both hyperconnectivity-hyperbinding and disinhibited feedback as potential underlying mechanisms, less research has explored the neuroanatomical basis of other forms of synesthesia. In the current study we investigated the white matter correlates of colored-music synesthesia. As these synesthetes report seeing colors upon hearing musical sounds, we hypothesized that they might show unique patterns of connectivity between visual and auditory association areas. We used diffusion tensor imaging to trace the white matter tracts in temporal and occipital lobe regions in 10 synesthetes and 10 matched non-synesthete controls. Results showed that synesthetes possessed hemispheric patterns of fractional anisotropy, an index of white matter integrity, in the inferior fronto-occipital fasciculus (IFOF), a major white matter pathway that connects visual and auditory association areas to frontal regions. Specifically, white matter integrity within the right IFOF was significantly greater in synesthetes than controls. Furthermore, white matter integrity in synesthetes was correlated with scores on audiovisual tests of the Synesthesia Battery, especially in white matter underlying the right fusiform gyrus. Our findings provide the first evidence of a white matter substrate of colored-music synesthesia, and suggest that enhanced white matter connectivity is involved in enhanced cross-modal associations.

Enhanced functional networks in absolute pitch.

Functional networks in the human brain give rise to complex cognitive and perceptual abilities. While the decrease of functional connectivity is linked to neurological and psychiatric disorders, less is known about the consequences of increased functional connectivity. One population that has exceptionally enhanced perceptual abilities is people with absolute pitch (AP) - an ability to categorize tones into pitch classes without reference. AP has been linked to exceptional talent as well as to psychiatric and neurological conditions. Here we show that AP possessors have increased functional activation during music listening, as well as increased degrees, clustering, and local efficiency of functional correlations, with the difference being highest around the left superior temporal gyrus. Our results provide the first evidence that increased functional connectivity in a small-world brain network is related to exceptional perceptual abilities in a healthy population.

Absolute Pitch and Synesthesia: Two Sides of the Same Coin? Shared and Distinct Neural Substrates of Music Listening.

People with Absolute Pitch can categorize musical pitches without a reference, whereas people with tone-color synesthesia can see colors when hearing music. Both of these special populations perceive music in an above-normal manner. In this study we asked whether AP possessors and tone-color synesthetes might recruit specialized neural mechanisms during music listening. Furthermore, we tested the degree to which neural substrates recruited for music listening may be shared between these special populations. AP possessors, tone-color synesthetes, and matched controls rated the perceived arousal levels of musical excerpts in a sparse-sampled fMRI study. Both APs and synesthetes showed enhanced superior temporal gyrus (STG, secondary auditory cortex) activation relative to controls during music listening, with left-lateralized enhancement in the APs and right-lateralized enhancement in the synesthetes. When listening to highly arousing excerpts, AP possessors showed additional activation in the left STG whereas synesthetes showed enhanced activity in the bilateral lingual gyrus and inferior temporal gyrus (late visual areas). Results support both shared and distinct neural enhancements in AP and synesthesia: common enhancements in early cortical mechanisms of perceptual analysis, followed by relative specialization in later association and categorization processes that support the unique behaviors of these special populations during music listening.